**Instruments Affected**: MIRI, NIRCam
Tested on MIRI Simulated data
This notebook is meant to test how the differing methods in the skymatch step affect photometry output in a catalog at the end of the source_catalog step. This is set up to test each of the skymatch methods (local, global, match, and global+match) and see if there are differences in photometry between them. The plot at the end of the notebook compare the total AB mag calculated in the source_catalog step by x position. If there are no differences, the magnitudes for each method should line up. If there are differences between them, it will show in different symbols not lining up.
This notebook runs the following steps only from calwebb_image3: skymatch, resample and source_catalog, checking outputs of each step along the way. The final result is the plot at the bottom of the notebook as described above.
The pipeline documentation can be found here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/skymatch/README.html
The pipeline code is available on GitHub: https://github.com/spacetelescope/jwst/tree/master/jwst/skymatch
The skymatch step can be used to compute sky values in a collection of input images that contain both sky and source signal. The sky values can be computed for each image separately or in a way that matches the sky levels amongst the collection of images so as to minimize their differences. This operation is typically applied before combining multiple images into a mosaic.
Background Matching algorithm page on the CalWG confluence page
https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Image+Background+Matching
This notebook processes level 2 images through the calwebb_image3 skymatch and resample steps and examines outputs for different sky method parameters.
1) Set up data path and image list file.
2) Set up association files.
3) Modify average backgrould level of input images.
4) Run skymatch step on images.
5) Run skymatch, resample and source_catalog for each skymethod parameter (local, global, match, global+match).
6) See if the differing skymatch methods affect the source photometry.
7) Testing other parameters (nclip, usigma, lsigma, lower, upper, skystat). (Still TBD.)
This notebook has been created to test how skymatch, resample and source_catalog work together (with subtract=True indicating that the subtraction is done in the skymatch step and subtract=False indicating that the subtraction is done in the resample step). There are displays of the combined noise image to see how well the backgrounds were subtracted in the full combined image.
These steps are set up with simulated MIRI F560W data of a crowded field with a variety of point sources and extended sources. There are four dither positions of the same field that are combined in the resample step. The notebook adds noise backgrounds to the field to help the pipeline testers have a better visualization of what the pipeline is doing with each set of parameters.
# Create a temporary directory to hold notebook output, and change the working directory to that directory.
from tempfile import TemporaryDirectory
import os
data_dir = TemporaryDirectory()
os.chdir(data_dir.name)
print(data_dir)
<TemporaryDirectory '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m'>
# Set up CRDS options
import os
if "CRDS_CACHE_TYPE" in os.environ:
if os.environ['CRDS_CACHE_TYPE'] == 'local':
os.environ['CRDS_PATH'] = os.path.join(os.environ['HOME'], 'crds', 'cache')
elif os.path.isdir(os.environ['CRDS_CACHE_TYPE']):
os.environ['CRDS_PATH'] = os.environ['CRDS_CACHE_TYPE']
The following packages are needed to run this notebook:
import pytest
from astropy.io import fits
from astropy.table import Table, vstack, unique, join
from astropy import table
from glob import glob
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import rcParams
%matplotlib inline
import jwst
import json
from jwst.skymatch import skymatch
from jwst.resample import ResampleStep
from jwst.pipeline import Image2Pipeline, Image3Pipeline
from jwst.skymatch.skyimage import SkyImage
from jwst.associations.lib.rules_level3_base import DMS_Level3_Base
from jwst.associations import asn_from_list
from jwst import datamodels
from jwst.datamodels import ImageModel
# Box download imports
from astropy.utils.data import download_file
from pathlib import Path
from shutil import move
from os.path import splitext
from ci_watson.artifactory_helpers import get_bigdata
jwst.__version__
'1.5.3'
def create_scatterplot(catalog_colx, catalog_coly, title=None):
''' Function to generate a generic scatterplot.
'''
fig = plt.figure(figsize=(8, 8))
ax = plt.subplot()
ax.scatter(catalog_colx,catalog_coly)
plt.xlabel(catalog_colx.name)
plt.ylabel(catalog_coly.name)
if title:
plt.title(title)
# Read in new dataset from Box
def get_box_files(file_list):
for box_url,file_name in file_list:
if 'https' not in box_url:
box_url = 'https://stsci.box.com/shared/static/' + box_url
downloaded_file = download_file(box_url)
if Path(file_name).suffix == '':
ext = splitext(box_url)[1]
file_name += ext
move(downloaded_file, file_name)
# Use F560W data set
file_urls = ['https://stsci.box.com/shared/static/g8ozd2wkdo9klrtkafpe9qegpzhpd66a.fits',
'https://stsci.box.com/shared/static/ym75nbqdi32t9wnbanm6s57itqixniyc.fits',
'https://stsci.box.com/shared/static/0bqh6ijsc6kd7cyv6gtnj9kvly9t885f.fits',
'https://stsci.box.com/shared/static/tywadpqcjhp9yxyv0sy91hc79red92xm.fits']
filenames = ['det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits',
'det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits',
'det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits',
'det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits']
# Test with F770W data set
#file_urls = ['https://stsci.box.com/shared/static/8573htf1p8mhk4e49z9b483dp975y3lz.fits',
# 'https://stsci.box.com/shared/static/p69h7uzlmqwlzf6kqqbf4lu9ibz28tc6.fits',
# 'https://stsci.box.com/shared/static/js8k6j20rek1oago057wxn08dhuticij.fits',
# 'https://stsci.box.com/shared/static/es253mc3m1sptusj9c9ctl1blidr1qdl.fits',
# 'https://stsci.box.com/shared/static/lpcgp7jkq6lmjj2lxpk63jz9u05qgk3a.cat',
# 'https://stsci.box.com/shared/static/15j6hkjc28zow4rjepob2t75dl887zs3.cat']
#file_names = ['det_image_seq1_MIRIMAGE_F770Wexp1_rate.fits',
# 'det_image_seq2_MIRIMAGE_F770Wexp1_rate.fits',
# 'det_image_seq3_MIRIMAGE_F770Wexp1_rate.fits',
# 'det_image_seq4_MIRIMAGE_F770Wexp1_rate.fits',
# 'input_sim_stars.cat',
# 'input_sim_galaxies.cat']
box_download_list = [(url,name) for url,name in zip(file_urls,filenames)]
get_box_files(box_download_list)
# Run Calwebb_image2 on output files from detector1
ratefiles = glob('*rate.fits')
print('There are ', len(ratefiles), ' images.')
callist = []
# cycle through files
for im in ratefiles:
pipe2 = Image2Pipeline()
rampfile = ImageModel(im)
filename = rampfile.meta.filename
# Set pipeline parameters
pipe2.save_results = True
pipe2.output_file = filename +'_cal.fits'
pipe2.resample.save_results = True
pipe2.suffix = None
calfile = pipe2.run(rampfile)
callist.append(calfile)
print(callist)
2022-06-20 21:50:41,045 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created. 2022-06-20 21:50:41,048 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created. 2022-06-20 21:50:41,050 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created. 2022-06-20 21:50:41,051 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created. 2022-06-20 21:50:41,053 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created. 2022-06-20 21:50:41,055 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
There are 4 images.
2022-06-20 21:50:42,211 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:42,225 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}}}
2022-06-20 21:50:42,237 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-06-20 21:50:42,586 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-06-20 21:50:42,589 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-06-20 21:50:42,590 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-06-20 21:50:42,591 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-06-20 21:50:42,592 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-06-20 21:50:42,593 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-06-20 21:50:42,599 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:50:42,601 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-06-20 21:50:42,602 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-06-20 21:50:42,604 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0725.fits'.
2022-06-20 21:50:42,608 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-06-20 21:50:42,609 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-06-20 21:50:42,610 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-06-20 21:50:42,611 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-06-20 21:50:42,612 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-06-20 21:50:42,613 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-06-20 21:50:42,614 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-06-20 21:50:42,615 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-06-20 21:50:42,617 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-06-20 21:50:42,618 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-06-20 21:50:42,619 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-06-20 21:50:42,620 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-06-20 21:50:42,622 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-06-20 21:50:42,622 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-06-20 21:50:42,686 - stpipe.Image2Pipeline - INFO - Processing product det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:50:42,687 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits> ...
2022-06-20 21:50:42,868 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:42,870 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-06-20 21:50:43,077 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-06-20 21:50:43,153 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.019532808 -0.016148354 0.021999350 0.015037197 359.990697195 0.017630137 359.987889369 -0.013569512
2022-06-20 21:50:43,154 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.019532808 -0.016148354 0.021999350 0.015037197 359.990697195 0.017630137 359.987889369 -0.013569512
2022-06-20 21:50:43,155 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-06-20 21:50:43,194 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.10/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-06-20 21:50:43,253 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-06-20 21:50:43,436 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:43,438 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-06-20 21:50:44,165 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:44,166 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:44,168 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:44,175 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:44,324 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-06-20 21:50:44,497 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:44,499 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-06-20 21:50:44,526 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-06-20 21:50:44,527 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-06-20 21:50:44,614 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-06-20 21:50:44,616 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-06-20 21:50:44,617 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-06-20 21:50:44,617 - stpipe.Image2Pipeline.photom - INFO - filter: F560W
2022-06-20 21:50:44,686 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-06-20 21:50:44,689 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-06-20 21:50:44,691 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-06-20 21:50:44,692 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 1.12701
2022-06-20 21:50:44,712 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-06-20 21:50:44,882 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:44,884 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:50:44,912 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:50:45,045 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-06-20 21:50:45,741 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:46,096 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:50:46,707 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:47,059 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:50:47,638 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:48,021 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:50:48,625 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:49,019 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.019529560 -0.016361051 0.022282746 0.015012151 359.990664200 0.017786867 359.987911014 -0.013586335
2022-06-20 21:50:49,151 - stpipe.Image2Pipeline.resample - INFO - Saved model in det_image_seq1_MIRIMAGE_F560Wexp1_i2d.fits
2022-06-20 21:50:49,152 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-06-20 21:50:49,153 - stpipe.Image2Pipeline - INFO - Finished processing product det_image_seq1_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:50:49,155 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-06-20 21:50:49,156 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0874.pmap
2022-06-20 21:50:49,330 - stpipe.Image2Pipeline - INFO - Saved model in det_image_seq1_MIRIMAGE_F560Wexp1_cal.fits
2022-06-20 21:50:49,331 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-06-20 21:50:49,337 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-06-20 21:50:49,339 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-06-20 21:50:49,341 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-06-20 21:50:49,343 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-06-20 21:50:49,344 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-06-20 21:50:49,346 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:50:49,593 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:49,599 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}}}
2022-06-20 21:50:49,608 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-06-20 21:50:49,618 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-06-20 21:50:49,621 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-06-20 21:50:49,622 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-06-20 21:50:49,623 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-06-20 21:50:49,624 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-06-20 21:50:49,625 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-06-20 21:50:49,628 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:50:49,631 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-06-20 21:50:49,632 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-06-20 21:50:49,635 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0725.fits'.
2022-06-20 21:50:49,637 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-06-20 21:50:49,638 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-06-20 21:50:49,638 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-06-20 21:50:49,639 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-06-20 21:50:49,640 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-06-20 21:50:49,641 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-06-20 21:50:49,642 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-06-20 21:50:49,643 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-06-20 21:50:49,646 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-06-20 21:50:49,646 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-06-20 21:50:49,647 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-06-20 21:50:49,647 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-06-20 21:50:49,648 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-06-20 21:50:49,649 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-06-20 21:50:49,709 - stpipe.Image2Pipeline - INFO - Processing product det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:50:49,712 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits> ...
2022-06-20 21:50:49,908 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:49,912 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-06-20 21:50:50,126 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-06-20 21:50:50,223 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.020171953 -0.017470149 0.022638494 0.013715402 359.991336339 0.016308341 359.988528513 -0.014891308
2022-06-20 21:50:50,224 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.020171953 -0.017470149 0.022638494 0.013715402 359.991336339 0.016308341 359.988528513 -0.014891308
2022-06-20 21:50:50,225 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-06-20 21:50:50,273 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.10/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-06-20 21:50:50,332 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-06-20 21:50:50,529 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:50,531 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-06-20 21:50:50,641 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:50,643 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:50,644 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:50,652 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:50,792 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-06-20 21:50:51,005 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:51,007 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-06-20 21:50:51,033 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-06-20 21:50:51,034 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-06-20 21:50:51,120 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-06-20 21:50:51,121 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-06-20 21:50:51,122 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-06-20 21:50:51,122 - stpipe.Image2Pipeline.photom - INFO - filter: F560W
2022-06-20 21:50:51,171 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-06-20 21:50:51,173 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-06-20 21:50:51,174 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-06-20 21:50:51,175 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 1.12701
2022-06-20 21:50:51,194 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-06-20 21:50:51,390 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:51,392 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:50:51,419 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:50:51,548 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-06-20 21:50:52,200 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:52,550 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:50:53,149 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:53,501 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:50:54,064 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:54,416 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:50:54,988 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:55,361 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.020168704 -0.017682847 0.022921890 0.013690355 359.991303344 0.016465072 359.988550158 -0.014908130
2022-06-20 21:50:55,478 - stpipe.Image2Pipeline.resample - INFO - Saved model in det_image_seq2_MIRIMAGE_F560Wexp1_i2d.fits
2022-06-20 21:50:55,480 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-06-20 21:50:55,480 - stpipe.Image2Pipeline - INFO - Finished processing product det_image_seq2_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:50:55,482 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-06-20 21:50:55,483 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0874.pmap
2022-06-20 21:50:55,652 - stpipe.Image2Pipeline - INFO - Saved model in det_image_seq2_MIRIMAGE_F560Wexp1_cal.fits
2022-06-20 21:50:55,654 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-06-20 21:50:55,658 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-06-20 21:50:55,660 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-06-20 21:50:55,661 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-06-20 21:50:55,663 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-06-20 21:50:55,664 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-06-20 21:50:55,666 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:50:55,908 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:55,913 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}}}
2022-06-20 21:50:55,921 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-06-20 21:50:55,925 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-06-20 21:50:55,928 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-06-20 21:50:55,929 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-06-20 21:50:55,929 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-06-20 21:50:55,930 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-06-20 21:50:55,932 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-06-20 21:50:55,934 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:50:55,936 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-06-20 21:50:55,937 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-06-20 21:50:55,939 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0725.fits'.
2022-06-20 21:50:55,941 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-06-20 21:50:55,942 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-06-20 21:50:55,943 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-06-20 21:50:55,943 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-06-20 21:50:55,944 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-06-20 21:50:55,945 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-06-20 21:50:55,945 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-06-20 21:50:55,946 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-06-20 21:50:55,948 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-06-20 21:50:55,948 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-06-20 21:50:55,949 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-06-20 21:50:55,950 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-06-20 21:50:55,950 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-06-20 21:50:55,951 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-06-20 21:50:56,011 - stpipe.Image2Pipeline - INFO - Processing product det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:50:56,012 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits> ...
2022-06-20 21:50:56,231 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:56,233 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-06-20 21:50:56,426 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-06-20 21:50:56,498 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.020403099 -0.019920916 0.022869640 0.011264635 359.991567485 0.013857575 359.988759659 -0.017342074
2022-06-20 21:50:56,499 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.020403099 -0.019920916 0.022869640 0.011264635 359.991567485 0.013857575 359.988759659 -0.017342074
2022-06-20 21:50:56,500 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-06-20 21:50:56,536 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.10/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-06-20 21:50:56,594 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-06-20 21:50:56,797 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:56,802 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-06-20 21:50:56,920 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:56,921 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:56,922 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:56,931 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:50:57,065 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-06-20 21:50:57,272 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:57,273 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-06-20 21:50:57,299 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-06-20 21:50:57,300 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-06-20 21:50:57,383 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-06-20 21:50:57,384 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-06-20 21:50:57,385 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-06-20 21:50:57,386 - stpipe.Image2Pipeline.photom - INFO - filter: F560W
2022-06-20 21:50:57,434 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-06-20 21:50:57,437 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-06-20 21:50:57,438 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-06-20 21:50:57,439 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 1.12701
2022-06-20 21:50:57,457 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-06-20 21:50:57,680 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:50:57,682 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:50:57,710 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:50:57,839 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-06-20 21:50:58,456 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:58,810 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:50:59,391 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:50:59,759 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:51:00,340 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:00,696 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:51:01,280 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:01,664 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.020399850 -0.020133613 0.023153036 0.011239589 359.991534491 0.014014305 359.988781304 -0.017358897
2022-06-20 21:51:01,793 - stpipe.Image2Pipeline.resample - INFO - Saved model in det_image_seq3_MIRIMAGE_F560Wexp1_i2d.fits
2022-06-20 21:51:01,794 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-06-20 21:51:01,795 - stpipe.Image2Pipeline - INFO - Finished processing product det_image_seq3_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:51:01,797 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-06-20 21:51:01,798 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0874.pmap
2022-06-20 21:51:01,969 - stpipe.Image2Pipeline - INFO - Saved model in det_image_seq3_MIRIMAGE_F560Wexp1_cal.fits
2022-06-20 21:51:01,970 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-06-20 21:51:01,977 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-06-20 21:51:01,978 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-06-20 21:51:01,980 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-06-20 21:51:01,982 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-06-20 21:51:01,983 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-06-20 21:51:01,985 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:51:02,261 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:51:02,266 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}}}
2022-06-20 21:51:02,273 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-06-20 21:51:02,283 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-06-20 21:51:02,284 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-06-20 21:51:02,285 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-06-20 21:51:02,286 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-06-20 21:51:02,287 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-06-20 21:51:02,287 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-06-20 21:51:02,289 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:51:02,291 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-06-20 21:51:02,291 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-06-20 21:51:02,294 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0725.fits'.
2022-06-20 21:51:02,296 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-06-20 21:51:02,296 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-06-20 21:51:02,297 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-06-20 21:51:02,298 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-06-20 21:51:02,298 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-06-20 21:51:02,299 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-06-20 21:51:02,300 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-06-20 21:51:02,300 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-06-20 21:51:02,302 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-06-20 21:51:02,303 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-06-20 21:51:02,304 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-06-20 21:51:02,304 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-06-20 21:51:02,305 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-06-20 21:51:02,305 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-06-20 21:51:02,358 - stpipe.Image2Pipeline - INFO - Processing product det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:51:02,359 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits> ...
2022-06-20 21:51:02,522 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:51:02,525 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-06-20 21:51:02,709 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-06-20 21:51:02,782 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.021938966 -0.019422071 0.024405507 0.011763480 359.993103352 0.014356419 359.990295526 -0.016843230
2022-06-20 21:51:02,783 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.021938966 -0.019422071 0.024405507 0.011763480 359.993103352 0.014356419 359.990295526 -0.016843230
2022-06-20 21:51:02,784 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-06-20 21:51:02,829 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.10/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-06-20 21:51:02,891 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-06-20 21:51:03,067 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:51:03,069 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-06-20 21:51:03,180 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:51:03,181 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:51:03,182 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:51:03,189 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-06-20 21:51:03,322 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-06-20 21:51:03,479 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:51:03,481 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-06-20 21:51:03,506 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-06-20 21:51:03,507 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-06-20 21:51:03,587 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-06-20 21:51:03,588 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-06-20 21:51:03,589 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-06-20 21:51:03,590 - stpipe.Image2Pipeline.photom - INFO - filter: F560W
2022-06-20 21:51:03,645 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-06-20 21:51:03,648 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-06-20 21:51:03,649 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-06-20 21:51:03,650 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 1.12701
2022-06-20 21:51:03,668 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-06-20 21:51:03,824 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits>,).
2022-06-20 21:51:03,826 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:51:03,852 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:51:03,979 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-06-20 21:51:04,577 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:04,937 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:51:05,505 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:05,857 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:51:06,431 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:06,787 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:51:07,373 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1023, 1031)
2022-06-20 21:51:07,747 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021935717 -0.019634769 0.024688903 0.011738433 359.993070357 0.014513150 359.990317171 -0.016860052
2022-06-20 21:51:07,868 - stpipe.Image2Pipeline.resample - INFO - Saved model in det_image_seq4_MIRIMAGE_F560Wexp1_i2d.fits
2022-06-20 21:51:07,869 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-06-20 21:51:07,870 - stpipe.Image2Pipeline - INFO - Finished processing product det_image_seq4_MIRIMAGE_F560Wexp1_rate.fits_cal
2022-06-20 21:51:07,872 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-06-20 21:51:07,872 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0874.pmap
2022-06-20 21:51:08,035 - stpipe.Image2Pipeline - INFO - Saved model in det_image_seq4_MIRIMAGE_F560Wexp1_cal.fits
2022-06-20 21:51:08,036 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
[[<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F560Wexp1_cal.fits>], [<ImageModel(1024, 1032) from det_image_seq2_MIRIMAGE_F560Wexp1_cal.fits>], [<ImageModel(1024, 1032) from det_image_seq3_MIRIMAGE_F560Wexp1_cal.fits>], [<ImageModel(1024, 1032) from det_image_seq4_MIRIMAGE_F560Wexp1_cal.fits>]]
#input_files=[]
#input_files=[input_file1,input_file2,input_file3,input_file4]
imlist1=['det_image_seq1_MIRIMAGE_F560Wexp1_cal.fits','det_image_seq2_MIRIMAGE_F560Wexp1_cal.fits','det_image_seq3_MIRIMAGE_F560Wexp1_cal.fits','det_image_seq4_MIRIMAGE_F560Wexp1_cal.fits']
# Look at one image
im_file = ImageModel('det_image_seq1_MIRIMAGE_F560Wexp1_cal.fits')
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=5)
plt.colorbar()
<matplotlib.colorbar.Colorbar at 0x7f3644f0c640>
Set parameters to use both in pipeline runs and for regions to examine for background area stats.
For MIRI, the FWHM values are dependent on filter and should be set using the table below:
| Filter | FWHM |
|---|---|
| F560W | 1.636 |
| F770W | 2.187 |
| F1000W | 2.888 |
| F1130W | 3.318 |
| F1280W | 3.713 |
| F1500W | 4.354 |
| F1800W | 5.224 |
| F2100W | 5.989 |
| F2550W | 7.312 |
| F2550WR | 7.312 |
# Choose region of image to examine for stats (preferably no bright sources)
xmin = 700
xmax = 750
ymin = 400
ymax = 450
# parameters for calwebb_image3 runs
snr = 5
fwhm = 1.7
The level three pipeline relies on an association file to specify which files are to be combined and provide the output file name.
# use asn_from_list to create association table
cal_list=imlist1
asn = asn_from_list.asn_from_list(cal_list, rule=DMS_Level3_Base, product_name='skymatch_combined.fits')
# use this if you need to add non'science' exposure types
#asn['products'][0]['members'][1]['exptype'] = 'background'
#asn['products'][0]['members'][2]['exptype'] = 'sourcecat'
# dump association table to a .json file for use in image3
with open('skymatch_asnfile.json', 'w') as fp:
fp.write(asn.dump()[1])
skymatch_json_file='skymatch_asnfile.json'
json_file = skymatch_json_file
file_list = []
file_list2 = []
with open(json_file) as json_data:
d = json.load(json_data)
members = d['products'][0]['members']
for item in np.arange(0,len(members)):
file_list.append(members[item]['expname'])
file_list2.append(members[item]['expname'][:-5]+"_skymatch.fits")
asn2 = asn_from_list.asn_from_list(file_list2, rule=DMS_Level3_Base, product_name=d['products'][0]['name'])
# use this if you need to add non'science' exposure types
#asn['products'][0]['members'][1]['exptype'] = 'background'
#asn['products'][0]['members'][2]['exptype'] = 'sourcecat'
# dump association table to a .json file for use in image3
with open('skymatch_asnfile2.json', 'w') as fp:
fp.write(asn2.dump()[1])
skymatch_json_file2='skymatch_asnfile2.json'
infile01_1 = imlist1[0]
infile01_2 = imlist1[1]
infile02_1 = imlist1[2]
infile02_2 = imlist1[3]
img01_1 = datamodels.open(infile01_1)
img01_2 = datamodels.open(infile01_2)
img02_1 = datamodels.open(infile02_1)
img02_2 = datamodels.open(infile02_2)
data01_1 = img01_1.data
data01_2 = img01_2.data
data02_1 = img02_1.data
data02_2 = img02_2.data
#data01_1[data01_1<=0.3]=np.nan
#data01_2[data01_2<=0.3]=np.nan
#data02_1[data02_1<=0.3]=np.nan
#data02_2[data02_2<=0.3]=np.nan
data01_1_orig = np.copy(img01_1.data)
data01_2_orig = np.copy(img01_2.data)
data02_1_orig = np.copy(img02_1.data)
data02_2_orig = np.copy(img02_2.data)
# check mean values of background
print('Mean:', np.nanmean(data01_1_orig[data01_1_orig<=4.0]),' Standard deviation: ',np.std(data01_1_orig[data01_1_orig<=4.0]))
print('Mean:', np.nanmean(data01_2_orig[data01_2_orig<=4.0]),' Standard deviation: ',np.std(data01_2_orig[data01_2_orig<=4.0]))
print('Mean:', np.nanmean(data02_1_orig[data02_1_orig<=4.0]),' Standard deviation: ',np.std(data02_1_orig[data02_1_orig<=4.0]))
print('Mean:', np.nanmean(data02_2_orig[data02_2_orig<=4.0]),' Standard deviation: ',np.std(data02_2_orig[data02_2_orig<=4.0]))
Mean: 0.9239596 Standard deviation: 0.41895863 Mean: 0.92317677 Standard deviation: 0.416928 Mean: 0.92502546 Standard deviation: 0.4192956 Mean: 0.9260735 Standard deviation: 0.4211552
# creating a background image with specified mean and gaussian noise with sigma = 1.0
bkg_img_noise_neg2 = np.random.normal(-2,2*0.2,data01_1.shape)
bkg_img_noise_2 = np.random.normal(2,2*0.2,data01_1.shape)
bkg_img_noise_3 = np.random.normal(3,3*0.2,data01_1.shape)
bkg_img_noise_4 = np.random.normal(4,4*0.2,data01_1.shape)
bkg_img_noise_5 = np.random.normal(5,5*0.2,data01_1.shape)
bkg_img_noise_7 = np.random.normal(7,7*0.2,data01_1.shape)
Add offset noise to the background of the images for easier background visualization
# adding the new background with specified mean and gaussian noise (above) to image
img01_1.data=bkg_img_noise_2 + img01_1.data
img01_2.data=bkg_img_noise_3 + img01_2.data
img02_1.data=bkg_img_noise_5 + img02_1.data
img02_2.data=bkg_img_noise_2 + img02_2.data
# checking the mean and standard deviations of the new background values
print('Mean:',np.nanmean(img01_1.data),' Standard deviation: ',np.nanstd(img01_1.data))
print('Mean:',np.nanmean(img01_2.data),' Standard deviation: ',np.nanstd(img01_2.data))
print('Mean:',np.nanmean(img02_1.data),' Standard deviation: ',np.nanstd(img02_1.data))
print('Mean:',np.nanmean(img02_2.data),' Standard deviation: ',np.nanstd(img02_2.data))
Mean: 3.7071698 Standard deviation: 14.163736 Mean: 4.698995 Standard deviation: 14.101952 Mean: 6.7133923 Standard deviation: 14.378862 Mean: 3.7183087 Standard deviation: 14.354485
img01_1.save(file_list2[0],overwrite=True)
img01_2.save(file_list2[1],overwrite=True)
img02_1.save(file_list2[2],overwrite=True)
img02_2.save(file_list2[3],overwrite=True)
'det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits'
# Look at one image
im_file = ImageModel(file_list2[0])
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
<matplotlib.colorbar.Colorbar at 0x7f3644798bb0>
Run the pipeline on the data for different sets of parameters
Notes: ‘local’: compute sky background values of each input image or group of images (members of the same “exposure”). A single sky value is computed for each group of images. This method simply computes the mean/median/mode/etc. value of the “sky” separately in each input image. This will resulted in subtracted background being near zero in the combined image, as each image has it's individual background subtracted.
match_down specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True)
'Subtract' specifies whether the computed sky background values are to be subtracted from the images in the skymatch step. (Default = False) Currently, if not done in skymatch, the subtraction is performed in resample.
# skymatch, local, subtract= False, all 4 images used
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='local', subtract=False,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skipping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'local' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = False # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:51:11,534 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:51:11,536 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:51:11,538 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:51:11,540 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:51:11,542 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:51:11,543 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:51:11,545 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:51:11,694 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:51:11,700 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'local', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:51:11,847 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:51:11,863 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:51:11,865 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:51:11,867 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:51:11,868 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:51:12,543 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:51:12,545 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:51:12,546 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:51:12,553 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:51:12,695 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:51:12,697 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'local', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:51:12,807 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:12,808 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:51:12.807485
2022-06-20 21:51:12,809 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:12,810 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'local'
2022-06-20 21:51:12,810 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-06-20 21:51:12,811 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:12,812 - stpipe.Image3Pipeline.skymatch - INFO - ---- Sky values computed per image and/or image groups.
2022-06-20 21:51:12,892 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635
2022-06-20 21:51:12,893 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 4.0126
2022-06-20 21:51:12,894 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 6.03289
2022-06-20 21:51:12,895 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.03724
2022-06-20 21:51:12,895 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:12,896 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:51:12.895681
2022-06-20 21:51:12,896 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:00.088196
2022-06-20 21:51:12,897 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:13,188 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:51:13,444 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:51:13,707 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:51:13,978 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:51:13,979 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:51:14,140 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:51:14,143 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:51:14,143 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:51:14,150 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:51:14,293 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:51:14,295 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:51:14,312 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:51:14,461 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:51:15,131 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:51:15,733 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:16,679 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:17,623 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:18,540 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:18,894 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:51:19,481 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:20,419 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:21,345 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:22,258 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:22,613 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:51:23,186 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:24,103 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:25,022 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:25,945 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:26,298 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:51:26,873 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:27,814 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:28,749 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:29,677 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:30,057 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:51:30,479 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:51:30,481 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:51:30,682 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:51:30,684 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:51:30,703 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:51:30,713 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:51:30,714 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:51:30,715 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:51:30,715 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:51:30,716 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:51:30,770 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:51:31,314 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 325 sources
2022-06-20 21:51:31,668 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:51:31,781 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:51:31,783 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:51:31,785 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:51:31,786 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
# Check values out of skymatch step. With subtract = False, values should be equal.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('Original and new levels should be equal. If subtract=False, skymatch should not subtract.')
print('Mean: original, new (local)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
Original and new levels should be equal. If subtract=False, skymatch should not subtract. Mean: original, new (local) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('The resample step should subtract the individual backgrounds found in the skymatch step.')
print('Since the local individual background level is subtracted from each image, final expected result is near zero.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
The resample step should subtract the individual backgrounds found in the skymatch step. Since the local individual background level is subtracted from each image, final expected result is near zero. Mean: 0.023842027
<matplotlib.colorbar.Colorbar at 0x7f364116dc00>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
cat_local = catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
Skymatch step should subtract the background. Resample background level should match the skymatch background level since subtraction has already been done
# skymatch, local, subtract = True, all 4 images used
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='local', subtract=True,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'local' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
#match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = True # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
#pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:51:33,382 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:51:33,383 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:51:33,385 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:51:33,387 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:51:33,388 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:51:33,390 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:51:33,392 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:51:33,563 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:51:33,570 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'local', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:51:33,715 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:51:33,718 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:51:33,720 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:51:33,721 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:51:33,722 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:51:34,426 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:51:34,429 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:51:34,430 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:51:34,436 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:51:34,609 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:51:34,611 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'local', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:51:34,722 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:34,723 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:51:34.722530
2022-06-20 21:51:34,724 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:34,725 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'local'
2022-06-20 21:51:34,725 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: ON
2022-06-20 21:51:34,726 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:34,727 - stpipe.Image3Pipeline.skymatch - INFO - ---- Sky values computed per image and/or image groups.
2022-06-20 21:51:34,808 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635 (old=0, delta=3.02635)
2022-06-20 21:51:34,811 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 4.0126 (old=0, delta=4.0126)
2022-06-20 21:51:34,812 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 6.03289 (old=0, delta=6.03289)
2022-06-20 21:51:34,814 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.03724 (old=0, delta=3.03724)
2022-06-20 21:51:34,815 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:34,815 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:51:34.815150
2022-06-20 21:51:34,816 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:00.092620
2022-06-20 21:51:34,816 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:35,105 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:51:35,372 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:51:35,630 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:51:35,891 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:51:35,892 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:51:36,079 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:51:36,081 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:51:36,082 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:51:36,088 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:51:36,259 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:51:36,261 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:51:36,279 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:51:36,431 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:51:37,099 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:51:37,688 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:38,600 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:39,520 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:40,434 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:40,786 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:51:41,355 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:42,270 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:43,181 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:44,089 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:44,442 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:51:45,013 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:45,925 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:46,834 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:47,747 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:48,103 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:51:48,671 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:49,606 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:50,531 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:51,443 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:51:51,816 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:51:52,234 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:51:52,235 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:51:52,433 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:51:52,435 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:51:52,453 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:51:52,462 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:51:52,463 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:51:52,464 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:51:52,465 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:51:52,466 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:51:52,516 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:51:53,060 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:51:53,391 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:51:53,503 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:51:53,505 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:51:53,506 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:51:53,507 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('With subtract=True, the new value should show that the background value was subtracted.')
print('Mean: original, new (local)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
With subtract=True, the new value should show that the background value was subtracted. Mean: original, new (local) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('The background value was subtracted in the skymatch step, no additional subtraction should be done here.')
print('This value should match the values in the new column above.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
The background value was subtracted in the skymatch step, no additional subtraction should be done here. This value should match the values in the new column above. Mean: 4.057631
<matplotlib.colorbar.Colorbar at 0x7f36407a3970>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
#Look at the sources found on the image
viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
plt.title('Sources found with Local method')
plt.imshow(viz2(im_file.data), origin='lower')
plt.colorbar()
plt.scatter(catalog['xcentroid'], catalog['ycentroid'],lw=1, s=10,color='red')
<matplotlib.collections.PathCollection at 0x7f364036bac0>
This calculates an average sky across all images and subtracts that average from all images. There should be a warning for users for this step that makes it clear that this should not be used on images with differences in background levels.
# skymatch, local
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='local', subtract=False,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'global' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
#match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = False # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
#pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:51:55,458 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:51:55,459 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:51:55,461 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:51:55,463 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:51:55,465 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:51:55,466 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:51:55,468 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:51:55,713 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:51:55,719 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:51:55,862 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:51:55,865 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:51:55,867 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:51:55,868 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:51:55,870 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:51:56,598 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:51:56,600 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:51:56,601 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:51:56,608 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:51:56,808 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:51:56,811 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:51:56,924 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:56,925 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:51:56.924301
2022-06-20 21:51:56,925 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:56,926 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'global'
2022-06-20 21:51:56,926 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-06-20 21:51:56,927 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:56,928 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing "global" sky - smallest sky value across *all* input images.
2022-06-20 21:51:57,009 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:57,010 - stpipe.Image3Pipeline.skymatch - INFO - "Global" sky value correction: 3.0263476342559947 [not converted]
2022-06-20 21:51:57,011 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635
2022-06-20 21:51:57,011 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635
2022-06-20 21:51:57,012 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635
2022-06-20 21:51:57,013 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635
2022-06-20 21:51:57,013 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:57,014 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:51:57.013622
2022-06-20 21:51:57,014 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:00.089321
2022-06-20 21:51:57,015 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:51:57,296 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:51:57,555 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:51:57,816 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:51:58,079 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:51:58,080 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:51:58,287 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:51:58,289 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:51:58,290 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:51:58,296 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:51:58,484 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:51:58,486 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:51:58,503 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:51:58,652 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:51:59,342 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:51:59,939 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:00,857 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:01,778 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:02,694 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:03,045 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:52:03,618 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:04,556 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:05,492 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:06,424 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:06,779 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:52:07,350 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:08,270 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:09,187 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:10,118 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:10,471 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:52:11,051 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:11,975 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:12,903 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:13,837 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:14,214 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:52:14,643 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:52:14,644 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:52:14,853 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:52:14,855 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:52:14,876 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:52:14,886 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:52:14,887 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:52:14,887 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:52:14,888 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:52:14,889 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:52:14,940 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:52:15,487 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:52:15,818 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:52:15,939 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:52:15,941 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:52:15,942 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:52:15,943 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('There should be no subtraction at this step, with subtraction = False.')
print('Mean: original, new (local)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
There should be no subtraction at this step, with subtraction = False. Mean: original, new (local) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('Global finds a single value for the background, and subtracts that value from all images.')
print('The value listed here should be the average of the subtracted sky values across images.')
print('The combined image will look messy as the same value was subtracted from the image with background = 5')
print('and from the images with background = 2. This shows the differing backgrounds strongly.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
Global finds a single value for the background, and subtracts that value from all images. The value listed here should be the average of the subtracted sky values across images. The combined image will look messy as the same value was subtracted from the image with background = 5 and from the images with background = 2. This shows the differing backgrounds strongly. Mean: 1.0312834
<matplotlib.colorbar.Colorbar at 0x7f363a696d10>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
cat_global = catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
#Look at the sources found on the image
viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
plt.title('Sources found with Global method')
plt.imshow(viz2(im_file.data), origin='lower')
plt.colorbar()
plt.scatter(catalog['xcentroid'], catalog['ycentroid'],lw=1, s=10,color='red')
<matplotlib.collections.PathCollection at 0x7f363a532560>
# skymatch, global, subtract = True
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='global', subtract=True,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'global' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
#match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = True # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
#pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:52:17,940 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:52:17,942 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:52:17,944 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:52:17,946 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:52:17,948 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:52:17,950 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:52:17,952 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:52:18,211 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:52:18,219 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:52:18,360 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:52:18,364 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:52:18,366 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:52:18,369 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:52:18,372 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:52:19,100 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:52:19,103 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:52:19,105 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:52:19,112 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:52:19,311 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:52:19,314 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:52:19,425 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,427 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:52:19.425877
2022-06-20 21:52:19,428 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,429 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'global'
2022-06-20 21:52:19,430 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: ON
2022-06-20 21:52:19,432 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,433 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing "global" sky - smallest sky value across *all* input images.
2022-06-20 21:52:19,514 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,515 - stpipe.Image3Pipeline.skymatch - INFO - "Global" sky value correction: 3.0263476342559947 [not converted]
2022-06-20 21:52:19,518 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635 (old=0, delta=3.02635)
2022-06-20 21:52:19,520 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635 (old=0, delta=3.02635)
2022-06-20 21:52:19,522 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635 (old=0, delta=3.02635)
2022-06-20 21:52:19,525 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.02635 (old=0, delta=3.02635)
2022-06-20 21:52:19,526 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,526 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:52:19.525999
2022-06-20 21:52:19,527 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:00.100122
2022-06-20 21:52:19,528 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:19,818 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:52:20,085 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:52:20,357 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:52:20,626 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:52:20,628 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:52:20,843 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:52:20,846 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:52:20,847 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:52:20,854 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:52:21,057 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:52:21,060 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:52:21,080 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:52:21,226 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:52:21,898 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:52:22,564 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:23,507 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:24,428 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:25,365 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:25,715 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:52:26,292 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:27,223 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:28,163 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:29,098 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:29,456 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:52:30,043 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:30,971 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:31,920 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:32,860 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:33,215 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:52:33,789 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:34,729 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:35,678 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:36,633 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:37,011 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:52:37,451 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:52:37,452 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:52:37,721 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:52:37,723 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:52:37,741 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:52:37,750 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:52:37,751 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:52:37,752 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:52:37,753 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:52:37,754 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:52:37,807 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:52:38,360 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:52:38,692 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:52:38,806 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:52:38,808 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:52:38,809 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:52:38,811 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('Global finds a single value for the background, and subtracts that value from all images.')
print('The values listed here should be the average of the subtracted sky values across images subtracted from each image.')
print('Mean: original, new (global)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
Global finds a single value for the background, and subtracts that value from all images. The values listed here should be the average of the subtracted sky values across images subtracted from each image. Mean: original, new (global) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('Global finds a single value for the background, and subtracts that value from all images.')
print('The value listed here should be the average of the subtracted sky values across images.')
print('The combined image will look messy as the same value was subtracted from the image with background = 5')
print('and from the images with background = 2. This shows the differing backgrounds strongly.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
Global finds a single value for the background, and subtracts that value from all images. The value listed here should be the average of the subtracted sky values across images. The combined image will look messy as the same value was subtracted from the image with background = 5 and from the images with background = 2. This shows the differing backgrounds strongly. Mean: 4.057631
<matplotlib.colorbar.Colorbar at 0x7f363a27d210>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
Based on whether match_down is set to True or False, Match will calculate the difference between the lowest background level or highest level (respectively), and subtract the difference between the calculated level and the 'matched' level.
This is the preferred default option as it subtracts the differences between the background levels, normalizing all of the backgrounds to a common level (either lowest or highest background), rather than subtracting off all of the background.
# skymatch, match down, subtract = True
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='match', match_down=True,subtract=True,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'match' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = True # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:52:40,186 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:52:40,188 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:52:40,190 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:52:40,192 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:52:40,194 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:52:40,196 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:52:40,199 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:52:40,472 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:52:40,479 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:52:40,622 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:52:40,626 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:52:40,628 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:52:40,630 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:52:40,632 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:52:41,366 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:52:41,369 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:52:41,371 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:52:41,380 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:52:41,592 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:52:41,594 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:52:41,710 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:41,711 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:52:41.710006
2022-06-20 21:52:41,712 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:41,712 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'match'
2022-06-20 21:52:41,713 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-06-20 21:52:41,713 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: ON
2022-06-20 21:52:41,714 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:41,714 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-06-20 21:52:44,433 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0 (old=0, delta=0)
2022-06-20 21:52:44,437 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.972833 (old=0, delta=0.972833)
2022-06-20 21:52:44,439 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 2.97689 (old=0, delta=2.97689)
2022-06-20 21:52:44,442 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.0259402 (old=0, delta=0.0259402)
2022-06-20 21:52:44,443 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:44,444 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:52:44.443148
2022-06-20 21:52:44,445 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.733142
2022-06-20 21:52:44,446 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:52:44,737 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:52:45,009 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:52:45,276 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:52:45,540 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:52:45,541 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:52:45,768 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:52:45,771 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:52:45,772 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:52:45,779 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:52:45,990 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:52:45,993 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:52:46,016 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:52:46,169 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:52:46,851 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:52:47,571 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:48,506 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:49,455 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:50,388 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:50,741 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:52:51,329 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:52,269 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:53,184 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:54,092 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:54,445 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:52:55,044 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:55,958 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:56,875 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:57,795 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:58,155 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:52:58,724 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:52:59,641 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:00,580 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:01,494 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:01,873 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:53:02,298 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:53:02,299 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:53:02,532 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:53:02,534 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:53:02,551 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:53:02,560 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:53:02,561 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:53:02,562 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:53:02,563 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:53:02,563 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:53:02,614 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:53:03,160 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:53:03,494 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:53:03,609 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:53:03,611 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:53:03,612 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:53:03,614 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('There should be no subtraction at this step, with subtraction = False.')
print('Mean: original, new (match down)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
There should be no subtraction at this step, with subtraction = False. Mean: original, new (match down) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('For the method match with match_down= True, the value of the background should match the minimum of the individual backgrounds.')
print('This value should match the values in the new column above. Subtraction was done in skymatch step.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
For the method match with match_down= True, the value of the background should match the minimum of the individual backgrounds. This value should match the values in the new column above. Subtraction was done in skymatch step. Mean: 4.057631
<matplotlib.colorbar.Colorbar at 0x7f3639dc1090>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
# skymatch, match up, subtract = True
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='match', match_down=False,subtract=True,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'match' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
match_down = False # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = True # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:53:05,005 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:53:05,006 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:53:05,008 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:53:05,010 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:53:05,011 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:53:05,013 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:53:05,015 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:53:05,311 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:53:05,318 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': False, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:53:05,463 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:53:05,466 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:53:05,468 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:53:05,470 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:53:05,471 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:53:06,234 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:53:06,237 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:53:06,238 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:53:06,244 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:53:06,481 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:53:06,484 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': False, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:53:06,611 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:06,612 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:53:06.611274
2022-06-20 21:53:06,613 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:06,613 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'match'
2022-06-20 21:53:06,614 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: UP
2022-06-20 21:53:06,615 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: ON
2022-06-20 21:53:06,615 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:06,616 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-06-20 21:53:09,418 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: -2.97689 (old=0, delta=-2.97689)
2022-06-20 21:53:09,421 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: -2.00406 (old=0, delta=-2.00406)
2022-06-20 21:53:09,423 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0 (old=0, delta=0)
2022-06-20 21:53:09,425 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: -2.95095 (old=0, delta=-2.95095)
2022-06-20 21:53:09,426 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:09,426 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:53:09.426312
2022-06-20 21:53:09,427 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.815038
2022-06-20 21:53:09,428 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:09,712 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:53:09,979 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:53:10,244 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:53:10,510 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:53:10,511 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:53:10,775 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:53:10,777 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:53:10,778 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:53:10,784 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:53:11,008 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:53:11,010 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:53:11,028 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:53:11,176 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:53:11,881 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:53:12,483 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:13,401 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:14,328 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:15,247 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:15,601 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:53:16,181 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:17,105 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:18,022 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:18,941 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:19,294 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:53:19,868 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:20,791 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:21,712 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:22,637 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:22,996 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:53:23,574 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:24,512 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:25,451 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:26,379 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:26,753 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:53:27,182 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:53:27,183 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:53:27,445 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:53:27,447 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:53:27,468 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:53:27,478 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:53:27,479 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:53:27,479 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:53:27,480 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:53:27,481 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:53:27,532 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:53:28,083 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:53:28,420 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:53:28,545 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:53:28,547 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:53:28,548 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:53:28,549 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('With match_down = False, match will match the backgrounds of all to the highest background.')
print('With subtract = True, the background matching is done at the skymatch step.')
print('Mean: original, new (match up)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
With match_down = False, match will match the backgrounds of all to the highest background. With subtract = True, the background matching is done at the skymatch step. Mean: original, new (match up) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('For the method match with match_down=False, the value of the background should match the maximum of the individual backgrounds')
print('This value should match the values in the new column above. Subtraction was done in skymatch step.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=7)
plt.colorbar()
For the method match with match_down=False, the value of the background should match the maximum of the individual backgrounds This value should match the values in the new column above. Subtraction was done in skymatch step. Mean: 4.057631
<matplotlib.colorbar.Colorbar at 0x7f363986c6d0>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'match' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = False # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:53:30,017 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:53:30,020 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:53:30,023 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:53:30,025 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:53:30,027 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:53:30,030 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:53:30,033 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:53:30,332 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:53:30,339 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:53:30,483 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:53:30,487 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:53:30,489 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:53:30,491 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:53:30,492 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:53:31,261 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:53:31,264 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:53:31,265 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:53:31,272 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:53:31,504 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:53:31,507 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:53:31,624 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:31,626 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:53:31.624889
2022-06-20 21:53:31,627 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:31,628 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'match'
2022-06-20 21:53:31,629 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-06-20 21:53:31,630 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-06-20 21:53:31,631 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:31,632 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-06-20 21:53:34,370 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0
2022-06-20 21:53:34,372 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.972833
2022-06-20 21:53:34,372 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 2.97689
2022-06-20 21:53:34,373 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.0259402
2022-06-20 21:53:34,374 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:34,375 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:53:34.374846
2022-06-20 21:53:34,376 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.749957
2022-06-20 21:53:34,377 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:34,666 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:53:34,939 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:53:35,206 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:53:35,465 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:53:35,467 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:53:35,721 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:53:35,723 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:53:35,725 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:53:35,733 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:53:35,965 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:53:35,967 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:53:35,987 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:53:36,136 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:53:36,822 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:53:37,441 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:38,358 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:39,270 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:40,187 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:40,537 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:53:41,119 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:42,045 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:42,960 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:43,867 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:44,222 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:53:44,787 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:45,713 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:46,636 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:47,560 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:47,915 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:53:48,499 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:49,435 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:50,508 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:51,599 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:53:51,986 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:53:52,456 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:53:52,457 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:53:53,022 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:53:53,025 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:53:53,044 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:53:53,056 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:53:53,056 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:53:53,057 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:53:53,058 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:53:53,058 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:53:53,116 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:53:53,698 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 325 sources
2022-06-20 21:53:54,042 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:53:54,187 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:53:54,190 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:53:54,191 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:53:54,193 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('With subtract=False, no subtraction should be done here.')
print('Mean: original, new (match down)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
With subtract=False, no subtraction should be done here. Mean: original, new (match down) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('With the match method, subtract=False and match_down=True, the background value should match to the lowest individual background.')
print('This value should match the minimum values in the new column above.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
With the match method, subtract=False and match_down=True, the background value should match to the lowest individual background. This value should match the minimum values in the new column above. Mean: 3.0573099
<matplotlib.colorbar.Colorbar at 0x7f3639431240>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
cat_match = catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
#Look at the sources found on the image
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
plt.title('Sources found with Match method')
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='gray', vmin=0, vmax=4)
plt.colorbar()
plt.scatter(catalog['xcentroid'], catalog['ycentroid'],lw=1, s=10,color='red')
<matplotlib.collections.PathCollection at 0x7f36392272b0>
The behavior of this step is that it subtracts all background. More documentation is needed on how exactly it gets to that point.
# skymatch, global+match, subtract = True
#jwst.skymatch.skymatch_step.SkyMatchStep.call(skymatch_json_file2, skymethod='global+match', match_down=True,subtract=True,output_file='MIRI',save_results=True)
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'global+match' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
#match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = True # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
#pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:53:56,413 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:53:56,415 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:53:56,417 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:53:56,419 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:53:56,421 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:53:56,423 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:53:56,424 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:53:57,144 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:53:57,151 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:53:57,318 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:53:57,323 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:53:57,325 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:53:57,326 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:53:57,327 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:53:58,584 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:53:58,587 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:53:58,588 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:53:58,595 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:53:59,224 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:53:59,229 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': True, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:53:59,408 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:59,409 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:53:59.408342
2022-06-20 21:53:59,410 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:59,411 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'global+match'
2022-06-20 21:53:59,411 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-06-20 21:53:59,412 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: ON
2022-06-20 21:53:59,413 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:53:59,413 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-06-20 21:54:02,203 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0 (old=0, delta=0)
2022-06-20 21:54:02,208 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.972833 (old=0, delta=0.972833)
2022-06-20 21:54:02,211 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 2.97689 (old=0, delta=2.97689)
2022-06-20 21:54:02,215 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.0259402 (old=0, delta=0.0259402)
2022-06-20 21:54:02,216 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:02,217 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing "global" sky - smallest sky value across *all* input images.
2022-06-20 21:54:02,304 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:02,305 - stpipe.Image3Pipeline.skymatch - INFO - "Global" sky value correction: 3.0113032313543875 [not converted]
2022-06-20 21:54:02,306 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:02,307 - stpipe.Image3Pipeline.skymatch - INFO - ---- Final (match+global) sky for:
2022-06-20 21:54:02,309 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.0113 (old=0, delta=3.0113)
2022-06-20 21:54:02,310 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.98414 (old=0.972833, delta=3.0113)
2022-06-20 21:54:02,312 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 5.98819 (old=2.97689, delta=3.0113)
2022-06-20 21:54:02,313 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.03724 (old=0.0259402, delta=3.0113)
2022-06-20 21:54:02,314 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:02,314 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:54:02.314096
2022-06-20 21:54:02,315 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.905754
2022-06-20 21:54:02,315 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:02,615 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:54:02,880 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:54:03,153 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:54:03,424 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:54:03,426 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:54:03,750 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:54:03,752 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:54:03,753 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:54:03,760 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:54:04,019 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:54:04,023 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:54:04,047 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:54:04,197 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:54:04,885 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:54:05,530 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:06,487 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:07,429 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:08,357 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:08,721 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:54:09,299 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:10,240 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:11,160 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:12,090 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:12,445 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:54:13,041 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:13,961 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:14,904 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:15,820 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:16,174 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:54:16,743 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:17,669 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:18,585 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:19,500 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:19,878 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:54:20,323 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:54:20,325 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:54:20,630 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:54:20,632 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:54:20,649 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:54:20,659 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:54:20,660 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:54:20,660 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:54:20,661 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:54:20,662 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:54:20,713 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:54:21,267 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 351 sources
2022-06-20 21:54:21,608 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:54:21,724 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:54:21,726 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:54:21,727 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:54:21,728 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('Mean: original, new (global+match)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
Mean: original, new (global+match) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
#print('Did resample subtract a background level, and what was it?')
print('This value should match the values in the new column above. Subtraction was done in skymatch step.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
This value should match the values in the new column above. Subtraction was done in skymatch step. Mean: 4.057631
<matplotlib.colorbar.Colorbar at 0x7f3638ce2ce0>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
# Test running just skymatch and resample and skiping other steps.
# Run Calwebb_image3 on the association table
# set any specific parameters
skymethod = 'global+match' # sky computation algorithm to be used. Allowed values: {local, global, match, global+match} (Default = global+match
#match_down = True # Specifies whether the sky differences should be subtracted from images with higher sky values (match_down = True) in order to match the image with the lowest sky or sky differences should be added to the images with lower sky values to match the sky of the image with the highest sky value (match_down = False). (Default = True
subtract = False # Specifies whether the computed sky background values are to be subtracted from the images. (Default = False
pipe3=Image3Pipeline()
pipe3.tweakreg.skip = True
pipe3.outlier_detection.skip = True
pipe3.skymatch.skymethod = skymethod
#pipe3.skymatch.match_down = match_down
pipe3.skymatch.subtract = subtract
pipe3.skymatch.output_file = 'MIRI'
pipe3.skymatch.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.source_catalog.snr_threshold = snr
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.save_results = True
# run Image3
image = pipe3.run(skymatch_json_file2)
print('Image 3 pipeline finished.')
2022-06-20 21:54:23,101 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-06-20 21:54:23,103 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-06-20 21:54:23,105 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-06-20 21:54:23,106 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-06-20 21:54:23,108 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-06-20 21:54:23,109 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-06-20 21:54:23,111 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-06-20 21:54:23,435 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('skymatch_asnfile2.json',).
2022-06-20 21:54:23,442 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-06-20 21:54:23,590 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-06-20 21:54:23,593 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-06-20 21:54:23,595 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-06-20 21:54:23,597 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-06-20 21:54:23,598 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-06-20 21:54:24,440 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-06-20 21:54:24,443 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 200, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 2.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'rshift', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2022-06-20 21:54:24,444 - stpipe.Image3Pipeline.tweakreg - INFO - Step skipped.
2022-06-20 21:54:24,451 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-06-20 21:54:24,705 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-06-20 21:54:24,707 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp6nb9ko6m/MIRI', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-06-20 21:54:24,822 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:24,824 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-06-20 21:54:24.822935
2022-06-20 21:54:24,824 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:24,825 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'global+match'
2022-06-20 21:54:24,826 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-06-20 21:54:24,826 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-06-20 21:54:24,827 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:24,828 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-06-20 21:54:27,590 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0
2022-06-20 21:54:27,592 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.972833
2022-06-20 21:54:27,592 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 2.97689
2022-06-20 21:54:27,593 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 0.0259402
2022-06-20 21:54:27,594 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:27,594 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing "global" sky - smallest sky value across *all* input images.
2022-06-20 21:54:27,674 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:27,675 - stpipe.Image3Pipeline.skymatch - INFO - "Global" sky value correction: 3.0113032415356806 [not converted]
2022-06-20 21:54:27,676 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:27,676 - stpipe.Image3Pipeline.skymatch - INFO - ---- Final (match+global) sky for:
2022-06-20 21:54:27,677 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq1_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.0113 (old=0, delta=3.0113)
2022-06-20 21:54:27,678 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq2_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.98414 (old=0.972833, delta=3.0113)
2022-06-20 21:54:27,678 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq3_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 5.98819 (old=2.97689, delta=3.0113)
2022-06-20 21:54:27,679 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=det_image_seq4_MIRIMAGE_F560Wexp1_cal_skymatch.fits. Sky background: 3.03724 (old=0.0259402, delta=3.0113)
2022-06-20 21:54:27,680 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:27,680 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-06-20 21:54:27.680164
2022-06-20 21:54:27,681 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.857229
2022-06-20 21:54:27,681 - stpipe.Image3Pipeline.skymatch - INFO -
2022-06-20 21:54:27,970 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_0_skymatch.fits
2022-06-20 21:54:28,228 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_1_skymatch.fits
2022-06-20 21:54:28,486 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_2_skymatch.fits
2022-06-20 21:54:28,746 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in MIRI_3_skymatch.fits
2022-06-20 21:54:28,747 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-06-20 21:54:29,034 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-06-20 21:54:29,036 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None}
2022-06-20 21:54:29,037 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-06-20 21:54:29,044 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-06-20 21:54:29,301 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-06-20 21:54:29,303 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None}
2022-06-20 21:54:29,328 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-06-20 21:54:29,487 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for skymatch_combined.fits
2022-06-20 21:54:30,170 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-06-20 21:54:30,765 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:31,682 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:32,596 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:33,513 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:33,867 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-06-20 21:54:34,441 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:35,372 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:36,313 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:37,248 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:37,605 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-06-20 21:54:38,208 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:39,146 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:40,066 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:40,981 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:41,334 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-06-20 21:54:41,903 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:42,838 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:43,781 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:44,703 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-06-20 21:54:45,080 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021880538 -0.020263552 0.024953987 0.014759122 359.990667339 0.017767981 359.987593889 -0.017254694
2022-06-20 21:54:45,512 - stpipe.Image3Pipeline.resample - INFO - Saved model in skymatch_combined_i2d.fits
2022-06-20 21:54:45,513 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-06-20 21:54:45,795 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from skymatch_combined_i2d.fits>,).
2022-06-20 21:54:45,797 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 1.7, 'snr_threshold': 5, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-06-20 21:54:45,815 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-06-20 21:54:45,826 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-06-20 21:54:45,827 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-06-20 21:54:45,827 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-06-20 21:54:45,828 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F560W
2022-06-20 21:54:45,829 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-06-20 21:54:45,880 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 3.76066
2022-06-20 21:54:46,424 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 325 sources
2022-06-20 21:54:46,745 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: skymatch_combined_cat.ecsv
2022-06-20 21:54:46,867 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in skymatch_combined_segm.fits
2022-06-20 21:54:46,869 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: skymatch_combined_segm.fits
2022-06-20 21:54:46,870 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-06-20 21:54:46,871 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
img01_1_new = datamodels.open('MIRI_0_skymatch.fits')
img01_2_new = datamodels.open('MIRI_1_skymatch.fits')
img02_1_new = datamodels.open('MIRI_2_skymatch.fits')
img02_2_new = datamodels.open('MIRI_3_skymatch.fits')
print('Mean: original, new (global+match)')
print('Mean:',np.nanmean(img01_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img01_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img01_2_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_1.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_1_new.data[ymin:ymax,xmin:xmax]))
print('Mean:',np.nanmean(img02_2.data[ymin:ymax,xmin:xmax]),',', np.nanmean(img02_2_new.data[ymin:ymax,xmin:xmax]))
Mean: original, new (global+match) Mean: 3.0139337 , 3.0139337 Mean: 4.0458083 , 4.0458083 Mean: 6.028047 , 6.028047 Mean: 3.0297823 , 3.0297823
# Look at combined image
im_file = ImageModel('skymatch_combined_i2d.fits')
print('Did resample subtract a background level, and what was it?')
#print('This value should match the values in the new column above. Subtraction was done in skymatch step.')
print('Mean:',np.nanmean(im_file.data[ymin:ymax,xmin:xmax]))
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
#viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
#plt.imshow(viz2(im_file.data), origin='lower')
plt.imshow(im_file.data, origin='lower', cmap='rainbow', vmin=0, vmax=4)
plt.colorbar()
Did resample subtract a background level, and what was it? Mean: 0.046006724
<matplotlib.colorbar.Colorbar at 0x7f3638a65cf0>
# Get name of output catalog
catfile = 'skymatch_combined_cat.ecsv'
print(catfile)
# Read in catalog
#catalog = Table.read(catfile)
catalog = table.Table.read(catfile, format='ascii', comment='#')
cat_global_match = catalog
# Plot AB mag values
create_scatterplot(catalog['label'], catalog['aper_total_abmag'],title='Total AB mag')
skymatch_combined_cat.ecsv
#Look at the sources found on the image
viz2 = LogStretch() + ManualInterval(0,10)
plt.figure(figsize=(20,20))
plt.title('Sources found with Global+Match method')
plt.imshow(viz2(im_file.data), origin='lower')
plt.colorbar()
plt.scatter(catalog['xcentroid'], catalog['ycentroid'],lw=1, s=10,color='red')
<matplotlib.collections.PathCollection at 0x7f3634b4ada0>
Determine whether the different methods give different photometric results. Ideally, the methods won't affect the output photometry, and the symbols should fall on top of each other for the sources plotted.
fig = plt.figure(figsize=(15, 15))
ax = plt.subplot()
plt.title('Photometry across methods')
plt.xlabel('x position')
plt.ylabel('aper_total_abmag')
ax.scatter(cat_local['xcentroid'],cat_local['aper_total_abmag'], c='black', marker='o', label='Local')
ax.scatter(cat_global['xcentroid'], cat_global['aper_total_abmag'], c='green', marker='v', label='Global')
ax.scatter(cat_match['xcentroid'], cat_match['aper_total_abmag'], c='blue', marker='x', label='Match')
ax.scatter(cat_global_match['xcentroid'], cat_global_match['aper_total_abmag'], c='magenta', marker='d', label='Global+Match')
ax.legend()
<matplotlib.legend.Legend at 0x7f36349d5fc0>
